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PREAMBLE 


The Standing Committee on Public Accounts held hearings on the Auditor 
General’s (Auditor’s) 2005 follow-up report on Children’s Mental Health 
Services administered by the Ministry of Children and Youth Services (section 
4.02 of his 2005 Annual Report) on February 16, 2006. The Committee has 
endorsed the Auditor’s findings and recommendations. 


This report flows from the Auditor’s 2005 Annual Report and the Committee’s 
February 2006 hearings. The Committee conducts hearings with ministry officials 
on selected audit reports to solicit current information on corrective measures 
taken following an audit and a ministry’s initial response. The Committee’s 
procedures include hearings and their deliberations, and most often, a report with 
new recommendations. Reports are tabled in the Legislature with the expectation 
that the auditee will provide a comprehensive response within a prescribed 
timeframe, indicating the corrective measures taken to ensure compliance. 


A Committee report provides background information in sections as found in the 
audit report, followed by an overview of the main findings resulting from the 
hearings and, as appropriate, new recommendations. 


In the case of Children’s Mental Health Services, the Committee held hearings in 
February 2004 and issued a report in July 2004 on the audit report that appeared 
in section 3.02 of the Auditor’s 2003 Annual Report. This report contains the 
Committee’s findings and recommendations as they relate to those areas of 
particular interest to Committee members. Hansard, the verbatim record of the 
hearings, should be consulted for the complete proceedings. 
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The Committee extends its appreciation to officials from the Ministry of Children 
and Youth Services for their attendance at the hearings. The Committee also 
acknowledges the assistance provided during the hearings by the Office of the 
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41. AUDIT OBJECTIVES AND MAIN FINDINGS 


The 2003 audit’s objectives were to assess whether the Ministry’s administrative 
procedures were adequate to ensure that: 


« the quality and outcome of services provided by the community-based 
agencies was monitored and assessed; and 


- transfer payments to agencies were reasonable and satisfactorily controlled.’ 





The 2003 audit concluded that the Ministry was not adequately monitoring and 
assessing the quality of services provided by the community-based agencies it 
funded. It found that the Ministry 


- had not established service quality standards and service evaluation 
criteria; 

« had not established waiting time standards that were reasonable and 
commensurate with individual children’s needs; 


¢ was not monitoring the extent and impact of lengthy waiting times; and 


¢ was not receiving or assessing information from agencies about service 
outcomes. 


The audit also found that funding decisions were not based on sufficiently 
detailed and relevant financial and operational information from agencies. Many 
of the 2003 audit report recommendations dealt with issues raised in a 1997 audit 
of the same program. 


2. COMMITTEE REQUEST FOR MINISTRY RESPONSE 


The Committee requests that the Ministry of Children and Youth Services provide 
the Committee Clerk with a written response within 120 calendar days of the 
tabling of this report with the Speaker of the Legislative Assembly, unless 
otherwise specified in a recommendation, as is the case in recommendations 2 and 
fe 


2.1. Committee Recommendations 


1. The Ministry of Children and Youth Services ensure that the continuum 
of services arising from its new policy framework responds to the needs of 
communities, parents and, most importantly, children with as much 
transparency as possible. 


2. The Ministry of Children and Youth Services report to the Committee 
on its established outcome measures and the reliability of baseline data for 
both the newly funded programs and the core services identified by its 
new policy framework, in an appearance before the Committee in the 
spring of 2007. 


3. The Ministry of Children and Youth Services report to the Committee 
on the template it has developed to be used by agencies and/or regional 
offices to ensure consistency in the recording of wait time and wait list 
data. The Ministry is asked to provide the Committee with a copy of the 
template prior to its appearance before the Committee in the spring of 
2007. 


4. The Ministry of Children and Youth Services provide the Committee 
with clarification of the phrase “7,165 clients (58%) received service 
within one month of referral” which appears above. [See page 9.] 


5. The Ministry of Children and Youth Services report to the Committee 
on the results of the second baseline data report after its release, 
particularly those related to wait times and outcome measures. 


6. The Ministry of Children and Youth Services ensure it is provided with 
and monitors waiting list and wait time data on a regular basis. 


7. The Ministry of Children and Youth Services provide an information 
systems update during its appearance before the Committee in the spring 
of 2007. 


8. The Ministry of Children and Youth Services consider the direct 
contractual agreement option with agencies providing services for the 
Intensive Early Intervention Program for Children with Autism. 


9. The Ministry of Children and Youth Services prepare a revised fully- 
loaded cost comparison between the direct-service and direct-funding 
options and report the outcome to the Committee. 


10. The Ministry of Children and Youth Services monitor the number of 
lost service hours. Where service hours are lost through no fault of a child 
or parent, the Committee expects the Ministry to ensure the hours will be 
made up or related funds recovered. 


3. OVERVIEW 


The Ministry of Children and Youth Services funds more than 250 agencies 
operating within the area of children’s mental health, including approximately 90 
child and youth mental health centres, hospital-based outpatient programs and a 
telepsychiatry service through the Hospital for Sick Children. It directly operates 
two child and youth mental health facilities: the Thistletown Regional Centre in 
Etobicoke, and the Child and Parent Resource Institute in London. In 2004/05, 
Ministry-funded programs provided mental health services to approximately 
153,000 children and youth.” 


The 2004 provincial budget allocated $25 million in new funds to child and youth 
mental health services. This grew to $38 million in 2005/06. The investment 
helped to create 113 new programs and expand 96 existing programs, and provide 
a 3% increase in funding for staff salaries at community agencies. (Funded 
services were determined through community planning tables.*) The 2005/06 
investment in mental health services for children and youth was $461.6 million.” 


AUDIT OBSERVATIONS AND RECOMMENDATIONS 
4. QUALITY OF SERVICE 


4.1 Auditor’s 2003 Annual Report 


The vast majority of funded programs had neither standards defining acceptable 
service nor criteria for evaluating service quality. This had resulted in the 
Ministry not having adequate assurance that funded programs met client needs or 
represented value for money spent. The Auditor found that service guidelines had 
been developed for several of the newly established programs. However, he noted 
that head office staff considered compliance with these guidelines to be 
mandatory while staff in regional offices, in most cases, did not. 


The Auditor recommended establishing standards for acceptable service quality 
and criteria for evaluating service quality. He also saw benefit in periodic 
evaluations of the quality of services and working with partner agencies to take 
corrective action where necessary. 


4.2 Committee’s 2004 Report 


The Committee recommended that the Ministry of Children and Youth Services 
provide it with 


an interim report on its progress in completing 
its review of children’s mental health services 
and developing new policy directions, as well as 
its identification of core services, and the 
development of service standards and 
guidelines.’ 


4.3 Ministry’s December 2005 Status Update 


Service standards allow for the effective evaluation of Ministry-funded services 
and the periodic assessment of their quality. Following on this, the Ministry has 
undertaken initiatives including the development of data and outcome measures 
an investment of $38 million in 2005/06 in new funding as a result of the 2004 
budget, and the development of a child and youth mental health policy 
framework. 
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Data and Outcome Measures 

In collaboration with the Provincial Centre of Excellence for Child and Youth 
Mental Health at the Children’s Hospital of Eastern Ontario (the Centre of 
Excellence), the Ministry has developed an evaluation framework for monitoring 
an outcome analysis of the new programs funded with the $38 million announced 
in the 2004 provincial budget. 


The framework has two phases; the first involves monitoring the implementation 
of the new programs and developing a program-specific methodology for 
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collecting data. Data collection tools have been developed and tested. Collection 
and analysis will begin in the spring of 2006. Phase two is a more detailed 
outcome evaluation of selected programs. The Centre of Excellence is leading the 
process to identify programs for the evaluation. The selection has begun. 


In addition, a baseline for wait times and client progress is being established, 
based on information obtained through the Brief Child and Family Phone 
Interview (BCFPI), and the Child and Adolescent Functional Assessment Scale 
(CAFAS). Both tools allow the Ministry to identify baseline data against which 
performance outcomes can be measured. The BCFPI is used in 108 agencies. The 
CAFAS is used in 107 community-based agencies and hospital-based outpatient 
programs. 


The mid- and long-range goals for the tools are to improve the size and reliability 
of data and to create “‘a culture of evidence-based service provision.” To do this, 
the Ministry is improving user training to enhance the update of the BCFPI and 
the CAFAS by making the products more user friendly and translating them into 
French. The Ministry is also continually evaluating and monitoring usage of the 
tools, as mandated in transfer payment service contracts and enforcement of 
compliance. 


Further work is underway to enhance the implementation and quality of data from 
both tools. The collected data guides and informs policy development, including 
information on wait times and outcome measures that will inform the 
development of the policy framework. 


New Investments — 2004 Budget 

The new and expanded services were funded as a result of a community-based 
planning process in the fall of 2004. Representatives from across the sector met to 
develop community plans to address local needs and service gaps. 


Policy Framework 

The new policy framework will identify guiding principles for the provision of 
services, service system goals across the continuum (e.g., health, education), 
levels of care and programs in a service continuum, and outcomes for future 
identification and measurement. 


The framework will be the foundation for the development of service standards 
and guidelines, which will permit the further development of outcome measures. 
Service standards and outcome measures will be evidence-based and reflect 
emerging practices to support a coordinated, balanced and accessible service 
system. The absence of a legislative or policy framework has limited the ability of 
the Ministry and service providers to establish clear service standards, definitions 
of care and system-wide outcome measures. 


In 2005, the Ministry and Children’s Mental Health Ontario (CMHO) undertook 
joint consultations on the framework with a range of stakeholders. In August 
2005, the Ministry distnbuted a background document on issues related to the 
framework to service providers and key stakeholders. From September to 





November 2005, the Ministry received input on the framework’s development. 
With CMHO, the Ministry had 14 community discussions with over 300 
representatives of key stakeholders and service providers. It also received 31 
written submissions. Inter-ministerial discussions were held with staff from the 
ministries of Children and Youth Services, Community and Social Services, the 
Attorney General, Education, and Health and Long-Term Care. 


Ongoing advice will be sought on the implementation of the framework and the 
development of evidence-based standards and guidelines through inter-ministerial 
and external consultation mechanisms. The framework was expected to be 
released in the spring of 2006.° 


4.4 Committee’s 2006 Hearings 


Data and Outcome Measures 
As of 2006/07, all funded agencies with licensed agreements will be required to 
use the BCFPI and the CAFAS.” 


Policy Framework 
The document to be released in the spring of 2006 will deal with vision, guiding 
principles, systems goals, and a broad description of a continuum of service.” 


Policy Framework - Consultations 
A variety of themes arose during the policy framework consultations. Ministry 
staff provided Committee members with an overview. 


One of the issues raised was consistency in service delivery expectations and 
availability in communities. Collaboration and integration at the local level were 
seen as helping to respond to capacity concerns, resource availability, the need for 
coordinated access, and the identification of appropriate continuums of service. 


Other themes included waiting lists and wait times, and the need for a more 
intensive focus on the specialized needs of those children considered to be the 
most in need. A widely heard comment was that providers are often dealing with 
clients with multiple and concurrent mental health problems involving more than 
one program. 


Reference to human resources issues (e.g., retention, recruitment, salaries) led to 
comment about training by Ministry staff. During the policy framework 
consultations, participants indicated the need to be trained by their agencies. 
Those involved in the planning tables referred to earlier in this report expressed a 
need for multi-disciplinary training sessions. 


Ministry staff saw training as a joint responsibility of regional offices and 
individual agencies. Agencies need to know what training is required to deliver 
their services. Once the policy framework is in place with its continuum of 
services and levels of care, the Ministry will need to play a leadership role and 
identify required competencies. The Centre of Excellence will be involved in 


training and competency development. It has already undertaken professional and 
clinical consultation sessions with agency staff.'! 


Supplementary Information 


The BCFPI is used at intake by licensed child and youth mental health agencies 
and other agencies that provide mental health services. The CAFAS 1s a clinical 
outcome instrument used by clinicians at the start of services and at intervals 
thereafter. Typically, clinical services occur several weeks or months after intake. 


As of May 2006, 120 agencies were using both the BCFPI and the CAFAS, nine 

used the CAFAS only and seven used the BCFPI only.'*, The Committee has also 
learned that the release of the policy framework has been delayed until the fall of 
2006 to facilitate review/comments by outside experts.” 


Committee Recommendations 


The Committee recommends that: 


1. The Ministry of Children and Youth Services ensure that the 
continuum of services arising from its new policy framework responds 
to the needs of communities, parents and, most importantly, children 
with as much transparency as possible. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response to this recommendation within 120 days 
of the tabling of this report in the Legislature. 


2. The Ministry of Children and Youth Services report to the 
Committee on its established outcome measures and the reliability of 
baseline data for both the newly funded programs and the core 
services identified by its new policy framework, in an appearance 
before the Committee in the spring of 2007. 


5. WAITING LISTS 


5.1 Auditor’s 2003 Annual Report 


The audit found that standards for access to services had not been established. 
With the exception of the autism program, information about waiting lists and 
times was not normally provided to the Ministry. A review of available 
information by the Auditor’s staff indicated that waiting times were often lengthy. 


The Auditor recommended establishing standards for access to service and 
assessing the extent to which standards were complied with. He also called for the 
development of strategies to monitor and remedy situations where waiting times 
were too lengthy. e 





5.2 Committee’s 2004 Report 
The Committee recommended that the Ministry of Children and Youth Services 


- direct its regional offices to request waiting list information from the agencies 
in their regions and to develop consolidated waiting lists for community-based 
and residential children’s mental health services which should ensure that 
children are not on more than one list; 


» require its regional offices to provide information about their consolidated 
waiting lists to the Ministry’s corporate offices on a quarterly basis; and 


« investigate the feasibility of developing a system whereby waiting lists are 
ensured to be accurate and reflective of the actual number of children waiting 
for services.'° 


5.3 Ministry’s December 2005 Status Update 


The Ministry is working with the Hospital for Sick Children and CMHO to 
develop baseline data for current waiting times, and to determine child and family 
functioning before and after service. 


The Ministry and CMHO have developed regional and provincial reports based 
on data collected by agencies using the BCFPI. CMHO began the process of 
collecting data on wait lists obtained from the BCFPI in 2004. This process 
includes developing clear definitions for wait lists, as well as training providers 
on how to collect and report information. The BCFPI has the capacity to provide 
critical data on “wait time” for services. The Ministry contracts with CMHO for 
the administration of the BCFPI. 


The BCFPI and the CAFAS also allow for the identification of baseline data, - 
against which performance outcomes can be measured. The first provincial report 
on baseline data showed that in 2004 the average waiting time for clients who 
were admitted to services province-wide was 36 days. Improvement in client 
functioning at exit of service in 2004 rated at 69.1%. 


The collection of waiting data is challenging for two reasons: the absence of a 
unique identifier makes it impossible to determine if clients are on multiple 
waiting lists; and the lack of standard terminology for a range of disorders and 
conditions for which stages of services may each have a wait time. 


CMHO conducted an analysis of the BCFPI data for 2004 and identified the 
variations in definitions that need to be corrected in order for the Ministry to 
obtain accurate data. In 2005/06, CMHO worked closely with agencies to develop 
common definitions for stages of services in order to increase consistency in 
application.’° 


5.4 Committee’s 2006 Hearings 


Agencies vary in size and sophistication in terms of their ability to use 
instruments like the BCFPI and the CAFAS, and other technologies. In 2005/06, 
the Ministry worked with CMHO to put better training in place for the staff who 
use the instruments. Software has been simplified to be more user-friendly and 
communities-of-practice have been established. Ministry staff felt the baseline 
data in the 2004 report was not good data. They were, however, confident that the 
data in the 2005 report would be better. 


Committee members were told that CMHO is collecting the BCFPI data on a 
quarterly basis. It is also talking to regional offices through planning tables and 
community-of-practice tables.' 


Waiting lists continue to be managed at the agency and regional levels. Ministry 
staff advised that this is appropriate for three reasons: the Ministry does not have 
the database capability; local management enables the local community to set its 
own priorities; and there is an increasing need for clinical judgements which the 
Ministry is unable to provide or comment on.'® 


Supplementary Information 

Wait time data from 92 of the largest child and youth mental health agencies in 
the province has been collected, “cleaned” and analyzed in order to develop a wait 
time baseline for the calendar year 2005. 


« The average wait time for 12,276 clients referred for and admitted to services 
was 44 days. 


¢ 7,165 clients (58%) received service within one month of referral. 


« Practice in this sector sees clients with the greatest need receiving services as 
quickly as possible. 


« The average wait time for 8,372 clients referred in 2004 and 2005 and still 
waiting for services on December 31, 2005 was 170 calendar days. 


Wait time data is shared with agencies, and the Ministry’s regional and corporate 
offices for ongoing monitoring and to guide corrective action or response when 
necessary. 


The Ministry sees wait time information as a more suitable planning indicator 
than wait list information. The latter does not differentiate between individuals 
who have just been referred for services and those who have been waiting for long 
periods of time. 


The BCFPI is used to track wait times and to obtain a profile of children and 
youth referred for services at the agency, regional and provincial levels. The 
clinical services measured by the CAFAS occur several weeks or months after 
intake.’ 
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Committee Recommendations 


The Committee acknowledges and appreciates regional and agency differences in 
the delivery of children’s mental health services. At the same time, it recognizes 
the need for consistency in the way in which demand for these services is 
recorded. 


The Committee therefore recommends that: 


3. The Ministry of Children and Youth Services report to the 
Committee on the template it has developed to be used by agencies 
and/or regional offices to ensure consistency in the recording of wait 
time and wait list data. The Ministry is asked to provide the 
Committee with a copy of the template prior to its appearance before 
the Committee in the spring of 2007. 


4. The Ministry of Children and Youth Services provide the 
Committee with clarification of the phrase “7,165 clients (58%) 
received service within one month of referral” which appears above. 


5. The Ministry of Children and Youth Services report to the 
Committee on the results of the second baseline data report after its 
release, particularly those related to wait times and outcome 
measures. 


6. The Ministry of Children and Youth Services ensure it is provided 
with and monitors waiting list and wait time data on a regular basis. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response to these recommendations within 120 
days of the tabling of this report in the Legislature. 


6. INFORMATION SYSTEMS 


6.1 Auditor’s 2003 Annual Report 


Financial and Service Information System 

A new management information system that captures annual and quarterly 
financial service information was implemented in August 1998. The audit found 
that it was not being used to analyze consistency of funding across the province or 
if the Ministry was receiving value for money for services provided by agencies. 
In those instances where it was being used for that type of analysis, the system 
was of limited usefulness. 


Autism Program Information System 

An information system that contains financial and client information for the nine 
regional autism programs was developed in 2000/01. The information is used to 
help develop new policies. The audit noted that it provides only consolidated 
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province-wide information and that Ministry staff said they were uncertain about 
the accuracy and completeness of the information.” 


6.2 Committee’s 2004 Report 
The Committee recommended that the Ministry of Children and Youth Services 


* report to the Committee within a year of the tabling of its report on the 
implementation of a compatible information system to track service and 
financial information for the use of all involved in the delivery of children’s 
mental health services; 


* provide its regional staff with the training necessary to make the Autism 
Program Information System (APIS) a more useful and reliable tool; and 


« ensure that the APIS is funded to a level that allows agency, regional and 
corporate staff to input and access data in an effective and efficient manner.” 


6.3 Ministry’s December 2005 Status Update 


Electronic Upload Files 

The Ministry planned to improve electronic update files by the late spring of 2006 
to allow for the automated transfer of data from agencies’ Excel spreadsheets to 
the Ministry’s Service Management Information System (SMIS). The initiative 
would facilitate information sharing between agencies and the Ministry. It would 
also improve the accuracy and reliability of SMIS data by reducing the potential 
for human error inherent in the current practice of manual data entry. 


Data Quality Assurance Tools 

In February 2005, the Ministry implemented additional data quality and assurance 
tools and methodologies to improve the integrity of data captured in the SMIS. 
These included an enhanced sign-off protocol requiring director-level sign-off on 
year-to-date data, and the provision of standardized exception and variance 
reports that allow for increased data review and validation of accuracy. 


Communication of Business Practices 

SMIS training had been and continued to be improved. In addition to the SMIS 
how-to training, the Ministry was educating staff on the validated business and 
financial practices that the SMIS supports. Through their training, staff were 
better able to identify data quality issues, and deal with them promptly and 
effectively. 


The enhanced training complemented the Ministry’s continuous efforts to 
communicate business and financial practices via the Business Managers 
Network, a formal structure comprised of regional and corporate finance and 
business managers, which facilitates coordinated and consistent business practices 
and decisions. 


12 





Improvements/Additions to SMIS 

A number of improvements based on the Auditor’s 2003 Annual Report had been 
implemented. They included a new chart of accounts introduced in SMIS when 
the Ministry was first created. In April 2004, the SMIS was recoded to operate 
with the Integrated Financial Information System (IFIS) and the new Ministry 
vendor structure.” 


6.4 Committee’s 2006 Hearings 


Ministry staff reported that they had taken several steps to ensure that their 
information systems provide sufficiently detailed, relevant and accurate 
information to allow for the monitoring of the cost-effectiveness of service 
delivery.” 


While the main information system is SMIS, staff referred to their dependence on 
an “old-fashioned” reporting relationship which sees information from the field 
transposed in regional offices. They felt that transpositional errors would be taken 
care of by future uploading to the SMIS database. 


Staff, particularly those in regional offices working in contract management and 
financial services, were being trained to ensure they had a clear understanding of 
expectations and requirements respecting field data. Effort had been taken to 
clearly define data elements. Service description standards, many of which had 
not been revised in 10 years or more, had been refined and standardized.” 


Supplementary Information 


Since the hearings, the Committee has been advised that approximately $3 million 
of the Ministry’s 2005/06 budget (about 0.09%) supported its information 
technology systems.” 


The Committee has also learned that the Ministry used the SMIS to manage most 
of its $2.9 billion in transfer payment funding in 2005/06. Several steps have been 
taken to improve the quality of data in the SMIS, among them 


« improving the electronic upload of Excel submission files; 


¢ improving data quality and assurance tools, including a regional director sign- 
off protocol on year-to-date reporting; and 


« automating the SMIS variance and exception reporting to regional staff. 


In November 2005, the Ministry established a team to review the transfer 
payment business cycle, as well as the SMIS application. Phase 1 of the project 
included a review of current processes, the Excel budget package, automation 
opportunities, and the SMIS interface. The team is also reviewing leading 
practices and transfer payment business models used by other jurisdictions, and 
recommendations for improving the transfer payment business process. A final 
report was issued on March 31, 2006. 
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The Ministry is reviewing the report’s recommendations and considering potential 
next steps, including the implementation of a number of short and longer term 
business process improvements to strengthen contract management and transfer 
payment accountability.”° 


Committee Recommendation 


The Committee is encouraged by the initiatives the Ministry has undertaken with 
respect to its information systems. 


The Committee therefore recommends that: 


7. The Ministry of Children and Youth Services provide an 
information systems update during its appearance before the 
Committee in the spring of 2007. 


7. MINISTRY RESPONSE TO REPORT ON REVIEW OF IEIP 


On April 8, 2004, the Committee passed a motion asking the Auditor to perform a 
review of the Intensive Early Intervention Program (IEIP) for Children with 
Autism. A report on that review was released on November 4, 2004.7’ On 
November 18 of that year, staff from the Ministry of Children and Youth Services 
appeared before the Committee to discuss the Auditor’s findings and 
recommendations.”® 


A status update on the Ministry’s commitments and undertakings since the release 
of the Auditor’s review, dated January 2006, was provided to the Committee prior 
to its hearings. During the course of those hearings, Ministry staff were 
questioned about four of the Auditor’s recommendations.” 


Supplementary Information 

As of March 31, 2006, 343 children were waiting to be assessed and 753 were 
waiting for Intensive Behavioural Intervention (IBI) services to begin.*’ The 
Ministry’s estimated expenditure on transfer payments for the autism portfolio in 
2005/06 was $95.168 million. Interim actuals were $95.064 million.”! 


7.1 Direct Contractual Agreements with Agencies 
The Auditor recommended that the Ministry 


should consider having a direct contractual 
agreement with each agency that provides 
services for the Intensive Early Intervention 
Program for Children with Autism.” 


Ministry’s January 2006 Current Status Update 


Regular meetings with lead service providers have explored current practices and 
the challenges of existing service delivery models, including subcontracting. 
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Based on these discussions, the Ministry will continue with the current model of 
subcontracting while continuing to consider enhancements that will improve 
monitoring and accountability requirements. 


Transfer Payment Budget Package Project 

The Transfer Payment Budget Packages used by transfer payment service 
providers are updated annually. Each package specifies the services to be 
delivered, includes financial information and sets out the Ministry’s expectations 
for performance and monitoring. 


In the fall of 2004, the Ministry established a working group to review the 
documents that relate to the autism intervention program (AIP) to improve their 
clarity, consistency and relevance. The changes minimize misinterpretation by 
providers and enable the Ministry to better use the data at both regional and 
provincial levels to monitor the progress of the AIP in meeting its targets. 


The working group met in the fall of 2005 to review the 2005/06 package. Some 
additional changes were made on the reporting capacity of service providers to 
further clarify the new data element included in 2004 defining children discharged 
from IBI and receiving transition support service. The group will continue to meet 
on an annual basis. *° 


Committee’s 2006 Hearings 


The Ministry has tried to be sensitive to the need for local delivery agents. 
Regional providers provide the overall direction for the program, as well as the 
training and the administrative link with the regional office. Subcontractors are 
trained and equally able to provide the service. The Ministry has also tried to 
ensure that the relationship between the primary service provider and the 
subcontractors is clarified so there is a clearer line of accountability. Regional 
service providers are ultimately accountable for the quality and quantity of 
service. They have clear obligations with respect to subcontractors on the delivery 
of services.** 


Supplementary Information 


The Ministry, in consultation with regional office staff, examined current 
practices, options and the impacts of changing service delivery methods, 
including subcontracting. Based on that examination, it decided to continue with 
the current model of subcontracting while continuing to consider enhancements 
that might improve monitoring and accountability requirements. 


The Ministry’s contracting processes support transfer payment accountability. 
Service contracts with individual providers establish and monitor accountability. 
Contracts outline the components of Ministry funding (e.g., budgetary and 
financial reporting requirements, program deliverables and outcomes, expected 
service volumes). The contracting process establishes service targets through 
negotiation with providers. It also collects detailed performance information on a 


ihe 


i * 


quarterly basis. The aim is to ensure agencies manage resources effectively and 
that value for money is achieved. 


New service description schedules were developed for the 2006/07 budget 
package. They are specific to the services to be delivered. The new schedules also 
require providers to develop plans for achieving approved service objectives. 


The terms and conditions of the transfer payment service contract include a non- 
assignment of services clause. Prior written approval from the province is 
required for the assignment of services to another party. Approval may be 
withheld by the province in its sole discretion or given subject to such conditions 
as it may impose. 


Each regional office has a program supervisor who is assigned lead responsibility 
for the autism portfolio. That individual inspects, supervises and advises providers 
through a variety of activities. These include 


- the negotiation of budgets and service contracts; 


« the analysis of program and financial information in quarterly reports from 
providers against the expectations set out in the service contract; 


¢ the analysis of the provider’s financial performance; 
« the investigation of serious incidents and/or complaints; and 


- regular contact.* 


Committee Recommendation 


The Committee recommends that: 


8. The Ministry of Children and Youth Services consider the direct 
contractual agreement option with agencies providing services for the 
Intensive Early Intervention Program for Children with Autism. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response to this recommendation within 120 days 
of the tabling of this report in the Legislature. 


7.2 Cost Variations Between Service Providers 
The Auditor recommended that 


where the costs of similar services vary 
significantly over time within or between 
individual service providers, the Ministry of 
Children and Youth Services should determine 
the reasons for such variances, and where 
necessary, take corrective action.*° 
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Ministry’s January 2006 Current Status Update 


The Ministry established the costing analysis project in March 2005. The project 
is using an “activity-based costing” model that is consistent with Ministry of 
Finance guidelines on costing government services. The model examines the cost 
to the province of the AIP. 


In the project’s first phase, the Ministry worked with the three lead service 
providers that were part of the Auditor General’s special report. The first step was 
defining the core functions of the AIP (e.g., intake services, eligibility 
determination, family and child supports, IBI, transition supports). The second 
step was the creation of a costing analysis tool that enabled service providers to 
assign financial and service data to each core function. To test the tool, the three 
lead service providers collected their best available service and financial data for 
2003/04. This process identified refinements that needed to be made so that the 
tool could be consistently used by all autism service providers. 


The Ministry is revising the costing analysis tool based on what has been learned 
from the pilot and to assess the feasibility of applying the tool to other autism 
service providers. The data collected will be reviewed to investigate cost 
variances, create a baseline for all function costs, examine the data in relation to 
quality of service indicators, and take any required corrective action. The scope of 
the project is limited to a comparison of cost data for program outputs. It does not 
address quality of service or program effectiveness.’ 


Committee’s 2006 Hearings 


Reference was made to the cost variances that occur between and within regions. 
Committee members were also told that amendments have been made to the 
costing analysis tool. The Ministry has notified all service providers that the tool 
will be sent out in the near future. Responses were expected within the following 
six months.*® 


7.3 Direct-Service Option versus Direct-Funding Option 
The Auditor recommended that the Ministry 


formally assess the relative advantages and 
disadvantages of the direct-service and the 
direct-funding options and determine whether 
the current mix of selected options provided 
facilitates the delivery of services to the largest 
number of children.’ 


Parents are given a choice of obtaining program services from a Ministry-funded 
provider (the direct-service option — DSO) or a qualified, private sector provider 
(the direct-funding option - DFO). With the DSO, instructor therapists are hired 
and paid for by the service agency, which also provides ancillary services such as 
parent training. Under the DFO, parents enter into a funding arrangement with the 
lead service provider in their region. (The DFO is not available in the northern 
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region.*°) Four times a year, they receive funding for the upcoming quarter which 
is based on an hourly rate multiplied by the number of program hours outlined in 
their child’s service plan. Some ancillary services are provided by the lead 
agency, but others are not. Those that are not provided are usually not funded.*! 
Initial assessments are undertaken by DSO providers regardless of the funding 
option eventually chosen by a parent.” 


Ministry’s January 2006 Current Status Update 


The costing analysis project referred to in 6.2 addresses this recommendation.” 


Committee’s 2006 Hearings 


The costing analysis project is the Ministry’s first effort to examine the key 
differences between the DSO and the DFO. While the analysis was recommended 
by the Auditor, the issue of differential costs had been raised in discussions with 
service providers and parents. (Approximately one third of parents elect to take 
the DFO.™*) 


For the purposes of the analysis, the key service components performed for the 
DSO and the DFO (e.g., intake, assessment, IBI as an entity itself, parent 
supports) were identified and mapped. The base costs for providing both options 
were identified in the absence of program administration and other factors. (The 
Ministry did not discount lost service hours.*”) 


The preliminary analysis found that the hourly DSO cost was in the area of $36, 
the hourly DFO cost in the area of $33. The Ministry wants to update this 
information as the figures used were from 2003/04. It also wishes to extend the 
analysis to the other six service providers. 


With the extension of the analysis to all service providers and a better 
understanding of the costs of care, Committee members were told that the 
Ministry will have to make decisions on how to standardize its approach to the 
two options to ensure the availability of the best possible services. If the costs of 
delivering services through the DFO are found to exceed the capability of parents, 
Ministry staff committed themselves to taking that into account.*° 


2004/05 Budget Allocation 

Ministry staff were asked why $2.7 million had been diverted from IBI to child 
welfare in 2004/05 when there were children on a waiting list for treatment and 
more waiting to be assessed to determine if they qualified for IBI. The $2.7 
million was described as an under-expenditure in the context of the additional $10 
million put into programs in 2004/05. The money was unspent at the end of the 
year. The Ministry made the decision to put it into child welfare, a legislated 
program. 


Capacity issues were cited as the principal reason for the choices made in 
2004/05. IBI expenditures were being tracked very carefully in 2005/06. Virtually 
all of that fiscal year’s allocation will be spent on its intended purpose.*® 
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Supplementary Information 

The Ministry provided the Committee with greater detail on the preliminary 
analysis of the hourly costs of the DSO and DFO. The project used an activity- 
based costing model consistent with the Ministry of Finance’s guidelines on 
costing government services. A template enabled the Ministry to collect detailed 
financial and service data on the core direct service functions of the AIP (e.g., 
intake services, determining eligibility). 


Preliminary data was based on the best available 2003/04 service and financial 
data from the three lead autism service providers. The template was designed to 
collect the most comparable direct service costs of the two delivery models. 


In addition to the average costs, the preliminary data identified that central 
administration costs for the providers were within the Ministry’s guideline of up 
to 10% for the total AIP budget in 2003/04. The preliminary data also identified 
that an average of 21% of the total cost of the AIP could be attributed to program 
administration. In the pilot template, administration costs included overall clinical 
and program management, administrative services, program-specific information 
technology services, staff time, and related costs not related to a specific child 
(e.g., training). 


Provincial data for 2005/06 will be collected from the nine lead service providers 
using a revised costing template. This information will enable the Ministry to 
review cost variances over time and among service providers.” 


Committee Recommendation 


The Committee recommends that: 


9. The Ministry of Children and Youth Services prepare a revised 
fully-loaded cost comparison between the direct-service and direct- 
funding options and report the outcome to the Committee. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response to this recommendation within 120 days 
of the tabling of this report in the Legislature. 


7.4 Lost Service Hours 
The Auditor recommended that the Ministry 


regularly receive and assess the extent of lost 
service hours for each service provider, take the 
necessary corrective action to minimize lost 
hours, and reassess its practice of allowing 
service providers to retain funding for 
undelivered service hours under the direct- 
service model.” 
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Ministry’s January 2006 Current Status Update 


The Ministry recognizes that, for various reasons, some children do not receive all 
of the service hours that have been clinically recommended. Several meetings 
have been held with lead service providers to discuss the reasons for and identify 
ways to reduce the number of cancelled service hours. 


All lead service providers have programs or procedures in place to reduce the 
number of hours cancelled due to staff availability. Examples include: 


¢ moving to a centre-based service delivery where staff to child ratios allow for 
more flexible staff reassignment; 


¢ introducing staff wellness programs to reduce the transmission of illnesses; 
and 


« policies to stagger staff vacations and coordinate agency training days. 


The lead service providers are making efforts to improve their ability to track 
cancellations due to staff and child availability. The Ministry will continue to 
work with them to monitor how programs are impacting the number of cancelled 
service hours.”! 


Committee Hearings 


The Ministry has not changed its policy regarding lost service hours. As they 
move toward a more standardized approach across all regions though, Ministry 
staff committed themselves to looking at a definition of lost service hours. Any 
definition would have to include reasonable guidelines for agencies and an 
accountable process. The principle would be that where it is through no fault of 
the parent or child that hours are lost, the Ministry would do its best to ensure 
that, if possible, time be made up. Agencies have been asked to track lost service 
hours, but the Ministry does not regularly receive that data.” 


Committee Recommendation 


The Committee recommends that: 


10. The Ministry of Children and Youth Services monitor the number 
of lost service hours. Where service hours are lost through no fault of 
a child or parent, the Committee expects the Ministry to ensure the 
hours will be made up or related funds recovered. 


The Committee requests that the Ministry provide the Committee 
Clerk with a written response to this recommendation within 120 days 
of the tabling of this report in the Legislature. 
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7.5 Evaluation of IEIP/AIP 
Ministry’s January 2006 Current Status Update 


Included with the Ministry’s responses to the Auditor’s recommendations 
concerning DSO versus DFO and lost service hours was reference to Dr. 
Adrienne Perry of York University. She had been contracted to conduct an 
analysis of the historical information identified through consultations with service 
providers. Dr. Perry’s review would systematically assess information currently 
available from regional service providers on children who had been involved in 
the AIP to date. It would produce a descriptive profile of the children, how they 
progressed and the factors that might be associated with that progress.» 


Committee Hearings 


Dr. Perry had been contracted to conduct a retrospective review, similar to work 
done for Surrey Place Centre which identified 89 children for whom there was 
entry, clinical and discharge-related data. The Centre, with help from Dr. Perry, 
did an extensive evaluation of program outcomes from a child and clinical 
perspective. Discussions with parents were included in that process. 


In terms of the Ministry’s project, Dr. Perry was sampling about 400 cases from 
across the province. Data collection had begun; cases for whom the Ministry had 
relevant data were being identified (i.e., intake information, clinical reports, 
discharge or transition data).>* 


Supplementary Information 


Following the hearings, the Ministry reported that because Dr. Perry’s work was a 
retrospective review of existing data, it was not possible to involve parents in her 
evaluation. Existing data on some family measures were being examined as part 
of the review, the scope of which precludes the collection of new data or 
information. The Ministry plans to proceed with an evaluation of the program. 
Families will have an opportunity to be involved in that process.” 
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